Summary

• Concurrent source fetching: Replace sequential source polling with asyncio.gather + Semaphore(5) for parallel fetching, significantly reducing total poll cycle time
• Batch database queries: Eliminate N+1 queries in _fetch_source (content existence checks), summarize_pending (source lookups), and _handle_first_posting_backfill (source lookups) by using batch operations
• Discord embed formatting: Switch content poster from plain text messages to structured Discord embeds with color-coded source types, timestamps, and clean bullet-point summaries
• Substack deduplication: Simplify SubstackAdapter by extending RSSAdapter directly, removing ~100 lines of duplicated RSS parsing logic
• Concurrent RSS discovery: Probe RSS feed paths concurrently instead of sequentially for faster feed detection

Changes

Pipeline performance (pipeline.py, repository.py)

• Add content_items_exist() batch method to Repository (single IN query vs N individual queries)
• Replace per-item content_item_exists() calls with batch lookup in _fetch_source
• Replace per-item get_source_by_id() calls with get_sources_by_ids() in summarize_pending and _handle_first_posting_backfill
• Add _is_due_for_poll() helper and _fetch_source_safe() error wrapper
• Fetch due sources concurrently with asyncio.gather and Semaphore(5)

Content poster (content_poster.py, content_posting.py)

• Replace format_message() with format_embed() returning discord.Embed
• Add _truncate_summary_at_bullet() to cleanly truncate summaries at bullet boundaries
• Update post_content() to send embeds instead of plain text
• Color-coded embeds by source type

Adapters (substack.py, rss_discovery.py)

• SubstackAdapter now extends RSSAdapter, removing duplicated parsing
• RSS discovery probes paths concurrently with asyncio.gather

Test plan

• All 606 tests pass
• ruff check . passes
• ruff format --check . passes
• mypy src/ passes
• 2 new tests for content_items_exist batch method
• Updated pipeline tests for batch mocks and concurrent fetching
• Updated content poster tests for embed format
• Updated RSS discovery tests for concurrent probing

@greptile